Pitfalls of Accurately Benchmarking Thermally Adaptive Chips

The performance of today’s chips varies over time due to active thermal management and energy conservation policies. Such changes in performance are necessary in thermally constrained systems that operate beyond sustainable thermal limits, such as Intel’s second generation Turbo Boost. This varying performance creates challenges in accurately benchmarking such systems. This paper gives examples of potential pitfalls of benchmarking thermally aware systems: extrapolating steady-state performance from a short run, comparing throughput across program runs of different lengths, and failing to consider recent system activity. We analyze situations in which these pitfalls can lead to measurement errors and discuss potential mitigation strategies. We experimentally demonstrate that simple methodological mistakes can result in measurement errors of 8% or more between steady-state and instantaneous throughput on a Turbo Boost-enabled system today. We conclude by discussing the implications of a widening gap between peak and sustainable performance in future systems, such as the recently proposed computational sprinting.